Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix device_type2sub_tsk_gph_builder_ #10563

Merged
merged 5 commits into from
Dec 26, 2024
Merged

fix device_type2sub_tsk_gph_builder_ #10563

merged 5 commits into from
Dec 26, 2024

Conversation

Flowingsun007
Copy link
Contributor

@Flowingsun007 Flowingsun007 commented Nov 12, 2024 •
edited
Loading

support multi devices training in graph mode(e.g. ascend npu)

@Flowingsun007 Flowingsun007 marked this pull request as ready for review November 13, 2024 07:08
@github-actions GitHub Actions
Copy link
Contributor

View latest API docs preview at: https://oneflow-staging.oss-cn-beijing.aliyuncs.com/docs/Oneflow-Inc/oneflow/pr/10563/

@github-actions GitHub Actions
Copy link
Contributor

Speed stats: 
GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.8ms (= 4376.1ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 59.1ms (= 5913.3ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.35 (= 59.1ms / 43.8ms)

OneFlow resnet50 time: 26.4ms (= 2643.8ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 37.5ms (= 3754.1ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.42 (= 37.5ms / 26.4ms)

OneFlow resnet50 time: 18.2ms (= 3642.5ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 37.3ms (= 7453.6ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 2.05 (= 37.3ms / 18.2ms)

OneFlow resnet50 time: 17.6ms (= 3513.0ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 32.2ms (= 6446.2ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.83 (= 32.2ms / 17.6ms)

OneFlow resnet50 time: 16.8ms (= 3361.0ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 29.1ms (= 5826.6ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.73 (= 29.1ms / 16.8ms)

OneFlow swin dataloader time: 0.201s (= 40.248s / 200, num_workers=1)
PyTorch swin dataloader time: 0.130s (= 25.999s / 200, num_workers=1)
Relative speed: 0.646 (= 0.130s / 0.201s)

OneFlow swin dataloader time: 0.057s (= 11.300s / 200, num_workers=4)
PyTorch swin dataloader time: 0.032s (= 6.461s / 200, num_workers=4)
Relative speed: 0.572 (= 0.032s / 0.057s)

OneFlow swin dataloader time: 0.030s (= 6.089s / 200, num_workers=8)
PyTorch swin dataloader time: 0.016s (= 3.260s / 200, num_workers=8)
Relative speed: 0.535 (= 0.016s / 0.030s)

❌ OneFlow resnet50 time: 49.4ms (= 4942.8ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 64.8ms (= 6483.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.31 (= 64.8ms / 49.4ms)

OneFlow resnet50 time: 36.9ms (= 3692.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 46.0ms (= 4600.6ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.25 (= 46.0ms / 36.9ms)

OneFlow resnet50 time: 27.8ms (= 5565.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 42.2ms (= 8448.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.52 (= 42.2ms / 27.8ms)

OneFlow resnet50 time: 25.0ms (= 5009.8ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.3ms (= 7658.7ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.53 (= 38.3ms / 25.0ms)

OneFlow resnet50 time: 25.1ms (= 5028.7ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 35.9ms (= 7180.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.43 (= 35.9ms / 25.1ms)

@Flowingsun007 Flowingsun007 enabled auto-merge (squash) November 18, 2024 02:10
@github-actions GitHub Actions
Copy link
Contributor

View latest API docs preview at: https://oneflow-staging.oss-cn-beijing.aliyuncs.com/docs/Oneflow-Inc/oneflow/pr/10563/

@github-actions GitHub Actions
Copy link
Contributor

Speed stats: 
GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.7ms (= 4372.1ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.6ms (= 5761.1ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.32 (= 57.6ms / 43.7ms)

OneFlow resnet50 time: 26.1ms (= 2614.3ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 36.9ms (= 3693.0ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.41 (= 36.9ms / 26.1ms)

OneFlow resnet50 time: 18.7ms (= 3733.2ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 36.3ms (= 7257.9ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.94 (= 36.3ms / 18.7ms)

OneFlow resnet50 time: 17.7ms (= 3535.6ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 31.7ms (= 6336.4ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.79 (= 31.7ms / 17.7ms)

OneFlow resnet50 time: 16.7ms (= 3340.4ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 28.7ms (= 5740.9ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.72 (= 28.7ms / 16.7ms)

OneFlow swin dataloader time: 0.202s (= 40.310s / 200, num_workers=1)
PyTorch swin dataloader time: 0.126s (= 25.231s / 200, num_workers=1)
Relative speed: 0.626 (= 0.126s / 0.202s)

OneFlow swin dataloader time: 0.055s (= 10.962s / 200, num_workers=4)
PyTorch swin dataloader time: 0.033s (= 6.527s / 200, num_workers=4)
Relative speed: 0.595 (= 0.033s / 0.055s)

OneFlow swin dataloader time: 0.030s (= 6.009s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.363s / 200, num_workers=8)
Relative speed: 0.560 (= 0.017s / 0.030s)

❌ OneFlow resnet50 time: 49.3ms (= 4932.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 64.3ms (= 6429.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.30 (= 64.3ms / 49.3ms)

OneFlow resnet50 time: 36.6ms (= 3659.2ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 47.2ms (= 4716.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.29 (= 47.2ms / 36.6ms)

OneFlow resnet50 time: 27.9ms (= 5577.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 39.2ms (= 7844.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.41 (= 39.2ms / 27.9ms)

OneFlow resnet50 time: 25.2ms (= 5042.5ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.8ms (= 7768.8ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.54 (= 38.8ms / 25.2ms)

OneFlow resnet50 time: 24.7ms (= 4935.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 36.3ms (= 7261.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.47 (= 36.3ms / 24.7ms)

@github-actions GitHub Actions
Copy link
Contributor

View latest API docs preview at: https://oneflow-staging.oss-cn-beijing.aliyuncs.com/docs/Oneflow-Inc/oneflow/pr/10563/

@Flowingsun007 Flowingsun007 requested review from oneflow-ci-bot and removed request for oneflow-ci-bot December 26, 2024 06:14
@github-actions GitHub Actions
Copy link
Contributor

Speed stats: 
GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.6ms (= 4360.5ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.3ms (= 5727.9ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.31 (= 57.3ms / 43.6ms)

OneFlow resnet50 time: 26.1ms (= 2608.5ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 37.4ms (= 3736.7ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.43 (= 37.4ms / 26.1ms)

OneFlow resnet50 time: 18.6ms (= 3729.9ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 35.6ms (= 7119.1ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.91 (= 35.6ms / 18.6ms)

OneFlow resnet50 time: 16.5ms (= 3300.0ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 32.5ms (= 6500.9ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.97 (= 32.5ms / 16.5ms)

OneFlow resnet50 time: 16.7ms (= 3336.9ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 29.0ms (= 5806.0ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.74 (= 29.0ms / 16.7ms)

OneFlow swin dataloader time: 0.200s (= 40.090s / 200, num_workers=1)
PyTorch swin dataloader time: 0.128s (= 25.510s / 200, num_workers=1)
Relative speed: 0.636 (= 0.128s / 0.200s)

OneFlow swin dataloader time: 0.054s (= 10.776s / 200, num_workers=4)
PyTorch swin dataloader time: 0.033s (= 6.542s / 200, num_workers=4)
Relative speed: 0.607 (= 0.033s / 0.054s)

OneFlow swin dataloader time: 0.031s (= 6.194s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.324s / 200, num_workers=8)
Relative speed: 0.537 (= 0.017s / 0.031s)

❌ OneFlow resnet50 time: 49.0ms (= 4897.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 65.0ms (= 6503.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.33 (= 65.0ms / 49.0ms)

OneFlow resnet50 time: 36.2ms (= 3620.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 48.5ms (= 4853.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.34 (= 48.5ms / 36.2ms)

OneFlow resnet50 time: 27.8ms (= 5551.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 39.8ms (= 7965.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.43 (= 39.8ms / 27.8ms)

OneFlow resnet50 time: 25.3ms (= 5055.3ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 39.2ms (= 7848.4ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.55 (= 39.2ms / 25.3ms)

OneFlow resnet50 time: 25.1ms (= 5029.2ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 36.2ms (= 7236.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.44 (= 36.2ms / 25.1ms)

@Flowingsun007 Flowingsun007 merged commit 998dc17 into master Dec 26, 2024
34 of 38 checks passed
@Flowingsun007 Flowingsun007 deleted the dev_fix_task_graph_builder branch December 26, 2024 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@clackhan clackhan clackhan approved these changes

@strint strint strint approved these changes

@oneflow-ci-bot oneflow-ci-bot Awaiting requested review from oneflow-ci-bot

Assignees
No one assigned
Labels
bug graph graph mode system
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Flowingsun007 @strint @clackhan